Today, an increasing number of Adaptive Deep Neural Networks (AdNNs) are being used on resource-constrained embedded devices. We observe that, similar to traditional software, redundant computation exists in AdNNs, resulting in considerable performance degradation. The performance degradation is dependent on the input and is referred to as input-dependent performance bottlenecks (IDPBs). To ensure an AdNN satisfies the performance requirements of resource-constrained applications, it is essential to conduct performance testing to detect IDPBs in the AdNN. Existing neural network testing methods are primarily concerned with correctness testing, which does not involve performance testing. To fill this gap, we propose DeepPerform, a scalable approach to generate test samples to detect the IDPBs in AdNNs. We first demonstrate how the problem of generating performance test samples detecting IDPBs can be formulated as an optimization problem. Following that, we demonstrate how DeepPerform efficiently handles the optimization problem by learning and estimating the distribution of AdNNs' computational consumption. We evaluate DeepPerform on three widely used datasets against five popular AdNN models. The results show that DeepPerform generates test samples that cause more severe performance degradation (FLOPs: increase up to 552\%). Furthermore, DeepPerform is substantially more efficient than the baseline methods in generating test inputs(runtime overhead: only 6-10 milliseconds).

I've been talking in recent posts about how our typical methods of testing AI systems are inadequate and potentially unsafe. In particular, I've complained that all of the headline-grabbing papers so far only do controlled experiments, so we don't how the AI systems will perform on real patients. Today I am going to highlight a piece of work that has not received much attention, but actually went "all the way" and tested an AI system in clinical practice, assessing clinical outcomes. They did an actual clinical trial! Big news … so why haven't you heard about it?

Test automation can result in reduction in cost and human effort. If the optimal policy, the course of actio ns taken, for the intended objective in a testing process could be learnt by the testing system (e.g., a smart tester agent), then it could be reused in similar situations, thus leading to higher efficiency, i.e., less computational time. Automating stress testing to find performance breaking points remains a challenge for complex software systems. Common approaches are mainly based on source code or system model analysis or use - case based techniques. However, source code or system models might not be avai lable at testing time. In this paper, we propose a self - adaptive fuzzy reinforcement learning - based performance (stress) testing framework (SaFReL) that enables the tester agent to learn the optimal policy for generating stress test case s leading to performance breaking point without access to performance model of the system under test. SaFReL learns the optimal policy through an initial learning, then reuses it during a transfer learning phase, while keeping the learning running in the long - term. Through multiple experiments on a simulated environment, we demonstrate that our approach generates the stress test case s for different programs efficiently and adaptively without access to performance models .

One of the things I am trying to do this year is some more technical posts (following up on some issues I have noticed at the intersection between medicine and machine learning). This is the first in a little mini-series on performance testing. Medical research has a different way of doing things, being more cautious about making claims and a bit more rigorous in justifying them, both of which are useful ideas to apply more broadly in machine learning (particularly at the applied end). While performance testing is often considered basic knowledge, one of my supervisors/colleagues is a bit of a ROC expert so I hope I can pass on some new ways of looking at things that are interesting even for some of the more knowledgeable folks around here.

Performance testing is a form of software testing that focuses on how a system running the system performs under a particular load. This is not about finding software bugs or defects. Performance testing should give developers the diagnostic information they need to eliminate bottlenecks. Types of performance testing for software To understand how software will perform on users' systems, there different types of performance tests that can be applied during software testing. This is non-functional testing, which is designed to determine the readiness of a system.